I want to start a bunch of subprocesses and write some lines to each of them with some configuration data, then write filenames to them to process, and then tell them there's no more work to do. Then I want to gather any results they produce. The key function is this:

proc doJobs(config: Config) =
  var processes = newSeq[Process](countProcessors())
  # Initialize all the processes
  for i in 0..processes.high:
    var process = startProcess(config.subcommand,
                               options={poStdErrToStdOut, poDemon})
    var stream = process.inputStream()
    stream.writeLine("*regex=", config.regex) # BAD
    stream.writeLine("*configured")
  # Dish out the work to all the processes: TODO do it "fairly"
  var i = 0
  for filename in getFilenames(config):
    var process = processes[i mod processes.len()]
    var stream = process.inputStream()
    stream.writeLine(filename)
  # Tell them there's no more work to do
  for i in 0..processes.high:
    processes[i].inputStream.writeLine("*finished")
  # Handle any results they produce
  while processes.len() > 0:
    var i = random(processes.len())
    var process = processes[i]
    if not process.running():
      process.close()
      processes.delete(i)
    else:
      for line in lines(process.outputStream()):
        echo line
Unfortunately this won't compile. The problem is at the line marked BAD which produces the error message:
searchcmd.nim(41, 11) Error: type mismatch: got (Stream, string, string)
but expected one of:
proc writeLine[Ty](f: File; x: varargs[Ty, `$`])
I also tried using inputHandle() instead of inputStream() but the only difference that made was in the type mismatch: got (FileHandle, string, string).

Can anyone advise?

2017-03-17 14:12:08

I can now get a variation to compile; but it doesn't actually work -- it crashes at runtime:

Traceback (most recent call last)
searchpdfscmd.nim(86)    searchpdfscmd
searchpdfscmd.nim(82)    main
searchpdfscmd.nim(50)    doJobs
osproc.nim(240)          inputHandle
SIGSEGV: Illegal storage access. (Attempt to read from nil?)
Error: execution of an external program failed: 'R:\searchpdfscmd\searchpdfscmd.exe v:\pdfs'

Here's the "improved" code (although it is much less nice than the original):

proc doJobs(config: Config) =
  var processes = newSeq[Process](countProcessors())
  # Initialize all the processes
  for i in 0..processes.high:
    var process = startProcess(config.subcommand,
                               options={poStdErrToStdOut, poDemon})
    var fh = process.inputHandle()
    var file: File
    if open(file, fh, mode=fmWrite):
      file.writeLine("*regex=", config.regex)
      file.writeLine("*configured")
  # Dish out the work to all the processes: TODO do it "fairly"
  var i = 0
  for filename in getFilenames(config):
    var process = processes[i mod processes.len()]
    var fh = process.inputHandle()
    var file: File
    if open(file, fh, mode=fmWrite):
      file.writeLine(filename)
  # Tell them there's no more work to do
  for i in 0..processes.high:
    var fh = processes[i].inputHandle()
    var file: File
    if open(file, fh, mode=fmWrite):
      file.writeLine("*finished")
  # Handle any results they produce
  while processes.len() > 0:
    var i = random(processes.len())
    var process = processes[i]
    if not process.running():
      process.close()
      processes.delete(i)
    else:
      var fh = processes[i].outputHandle()
      var file: File
      if open(file, fh):
        for line in lines(file):
          echo line

2017-03-17 17:24:12

You cannot use the handles with the file IO and expect things to work. osproc gives you inputStream() and outputStream() for a reason.

import osproc, streams

...
  for i in 0..processes.high:
    var process = startProcess(config.subcommand,
                               options={poStdErrToStdOut, poDemon})
    var fh = process.inputStream()
    fh.writeLine("*regex=", config.regex)
    fh.writeLine("*configured")
    # ensure data is transmitted!
    fh.flush()

2017-03-17 19:57:10

Your reply shows almost exactly how I did it at first!

However, I now realise that it failed to work because I hadn't imported streams, as your snippet shows, so thanks for that.

The import means that it now builds and runs. It does crash though, so I'll have to look into that, but it is a start.

2017-03-17 22:15:12