파워셸에서 N개의 병렬 작업 실행

파워셸에서 N개의 병렬 작업 실행

다음 파워셸 스크립트가 있습니다.

$list = invoke-sqlcmd 'exec getOneMillionRows' -Server...
$list | % {
    GetData $_ > $_.txt
    ZipTheFile $_.txt $_.txt.zip
    ...
}

스크립트 블록 실행 방법({ GetDatta $_ > $_.txt ....}) 제한된 최대 작업 수와 동시에(예: 한 번에 최대 8개의 파일을 생성할 수 있음)

사용자 "Start-Automating"이 게시한 것과 동일한 아이디어이지만, 예에서 다른 절을 누를 때 보류된 작업을 시작하지 않는 것에 대한 버그를 수정했습니다.

$servers = @('a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n')

foreach ($server in $servers) {
    $running = @(Get-Job | Where-Object { $_.State -eq 'Running' })
    if ($running.Count -ge 4) {
        $running | Wait-Job -Any | Out-Null
    }

    Write-Host "Starting job for $server"
    Start-Job {
        # do something with $using:server. Just sleeping for this example.
        Start-Sleep 5
        return "result from $using:server"
    } | Out-Null
}

# Wait for all jobs to complete and results ready to be received
Wait-Job * | Out-Null

# Process the results
foreach($job in Get-Job)
{
    $result = Receive-Job $job
    Write-Host $result
}

Remove-Job -State Completed

Start-Job cmdlet을 사용하면 백그라운드에서 코드를 실행할 수 있습니다.당신이 원하는 것을 하기 위해서는 아래의 코드와 같은 것이 작동해야 합니다.

foreach ($server in $servers) {
    $running = @(Get-Job | Where-Object { $_.State -eq 'Running' })
    if ($running.Count -le 8) {
        Start-Job {
             Add-PSSnapin SQL
             $list = invoke-sqlcmd 'exec getOneMillionRows' -Server...
             ...
        }
    } else {
         $running | Wait-Job
    }
    Get-Job | Receive-Job
}

이게 도움이 되길 바랍니다.

그것은 정말 쉬울 것입니다.Split-PipelinesplitPipeline 모듈의 cmdlet입니다.코드는 다음과 같이 단순하게 표시됩니다.

Import-Module SplitPipeline
$list = invoke-sqlcmd 'exec getOneMillionRows' -Server...
$list | Split-Pipeline -Count 8 {process{
    GetData $_ > $_.txt
    ZipTheFile $_.txt $_.txt.zip
    ...
}}

오래된 스레드이지만 이것이 도움이 될 수 있다고 생각합니다.

$List = C:\List.txt
$Jobs = 8

Foreach ($PC in Get-Content $List)
{
Do
    {
    $Job = (Get-Job -State Running | measure).count
    } Until ($Job -le $Jobs)

Start-Job -Name $PC -ScriptBlock { "Your command here $Using:PC" }
Get-Job -State Completed | Remove-Job
}

Wait-Job -State Running
Get-Job -State Completed | Remove-Job
Get-Job

실행 중인 작업의 양이 실행할 수 있는 "$jobs"의 양을 초과하면 "Do" 루프가 "forach"를 일시 중지합니다.나머지 작업이 완료되고 실패한 작업이 표시될 때까지 기다리는 것보다...

배경 직업이 답입니다.[System]을 사용하여 실행 대기열의 작업을 조절할 수도 있습니다.수집.대기열].이 주제에 대한 PowerShell 팀의 블로그 게시물은 다음과 같습니다. https://devblogs.microsoft.com/powershell/scaling-and-queuing-powershell-background-jobs/

대기열 방법을 사용하는 것이 백그라운드 작업을 조절하는 가장 좋은 방법일 수 있습니다.

멀티스레드 기능을 사용하고 개선하면 다음과 같이 사용할 수 있습니다.

$Script = {
    param($Computername)
    get-process -Computername $Computername
}

@('Srv1','Srv2') | Run-Parallel -ScriptBlock $Script

스크립트에 이 코드 포함

function Run-Parallel {
    <#
        .Synopsis
            This is a quick and open-ended script multi-threader searcher
            http://www.get-blog.com/?p=189#comment-28834
            Improove by Alban LOPEZ 2016

        .Description
            This script will allow any general, external script to be multithreaded by providing a single
            argument to that script and opening it in a seperate thread.  It works as a filter in the
            pipeline, or as a standalone script.  It will read the argument either from the pipeline
            or from a filename provided.  It will send the results of the child script down the pipeline,
            so it is best to use a script that returns some sort of object.

        .PARAMETER ScriptBlock
            This is where you provide the PowerShell ScriptBlock that you want to multithread.

        .PARAMETER ItemObj
            The ItemObj represents the arguments that are provided to the child script.  This is an open ended
            argument and can take a single object from the pipeline, an array, a collection, or a file name.  The
            multithreading script does it's best to find out which you have provided and handle it as such.
            If you would like to provide a file, then the file is read with one object on each line and will
            be provided as is to the script you are running as a string.  If this is not desired, then use an array.

        .PARAMETER InputParam
            This allows you to specify the parameter for which your input objects are to be evaluated.  As an example,
            if you were to provide a computer name to the Get-Process cmdlet as just an argument, it would attempt to
            find all processes where the name was the provided computername and fail.  You need to specify that the
            parameter that you are providing is the "ComputerName".

        .PARAMETER AddParam
            This allows you to specify additional parameters to the running command.  For instance, if you are trying
            to find the status of the "BITS" service on all servers in your list, you will need to specify the "Name"
            parameter.  This command takes a hash pair formatted as follows:

            @{"key" = "Value"}
            @{"key1" = "Value"; "key2" = 321; "key3" = 1..9}

        .PARAMETER AddSwitch
            This allows you to add additional switches to the command you are running.  For instance, you may want
            to include "RequiredServices" to the "Get-Service" cmdlet.  This parameter will take a single string, or
            an aray of strings as follows:

            "RequiredServices"
            @("RequiredServices", "DependentServices")

        .PARAMETER MaxThreads
            This is the maximum number of threads to run at any given time.  If ressources are too congested try lowering
            this number.  The default value is 20.

        .PARAMETER SleepTimer_ms
            This is the time between cycles of the child process detection cycle.  The default value is 200ms.  If CPU
            utilization is high then you can consider increasing this delay.  If the child script takes a long time to
            run, then you might increase this value to around 1000 (or 1 second in the detection cycle).

        .PARAMETER TimeOutGlobal
            this is the TimeOut in second for listen the last thread, after this timeOut All thread are closed, only each other are returned

        .PARAMETER TimeOutThread
            this is the TimeOut in second for each thread, the thread are aborted at this time

        .PARAMETER PSModules
            List of PSModule name to include for use in ScriptBlock

        .PARAMETER PSSapins
            List of PSSapin name to include for use in ScriptBlock

        .EXAMPLE
            1..20 | Run-Parallel -ScriptBlock {param($i) Start-Sleep $i; "> $i sec <"} -TimeOutGlobal 15 -TimeOutThread 5
        .EXAMPLE
            Both of these will execute the scriptBlock and provide each of the server names in AllServers.txt
            while providing the results to GridView.  The results will be the output of the child script.

            gc AllServers.txt | Run-Parallel $ScriptBlock_GetTSUsers -MaxThreads $findOut_AD.ActiveDirectory.Servers.count -PSModules 'PSTerminalServices' | out-gridview
    #>
    Param(
        [Parameter(ValueFromPipeline=$true,ValueFromPipelineByPropertyName=$true)]
            $ItemObj,
        [ScriptBlock]$ScriptBlock = $null,
        $InputParam = $Null,
        [HashTable] $AddParam = @{},
        [Array] $AddSwitch = @(),
        $MaxThreads = 20,
        $SleepTimer_ms = 100,
        $TimeOutGlobal = 300,
        $TimeOutThread = 100,
        [string[]]$PSSapins = $null,
        [string[]]$PSModules = $null,
        $Modedebug = $true
    )
    Begin{
        $ISS = [system.management.automation.runspaces.initialsessionstate]::CreateDefault()
        ForEach ($Snapin in $PSSapins){
            [void]$ISS.ImportPSSnapIn($Snapin, [ref]$null)
        }
        ForEach ($Module in $PSModules){
            [void]$ISS.ImportPSModule($Module)
        }
        $RunspacePool = [runspacefactory]::CreateRunspacePool(1, $MaxThreads, $ISS, $Host)
        $RunspacePool.CleanupInterval=1000
        $RunspacePool.Open()

        $Jobs = @()
    }
    Process{
        #ForEach ($Object in $ItemObj){
            if ($ItemObj){
                Write-Host $ItemObj -ForegroundColor Yellow
                $PowershellThread = [powershell]::Create().AddScript($ScriptBlock)

                If ($InputParam -ne $Null){
                    $PowershellThread.AddParameter($InputParam, $ItemObj.ToString()) | out-null
                }Else{
                    $PowershellThread.AddArgument($ItemObj.ToString()) | out-null
                }
                ForEach($Key in $AddParam.Keys){
                    $PowershellThread.AddParameter($Key, $AddParam.$key) | out-null
                }
                ForEach($Switch in $AddSwitch){
                    $PowershellThread.AddParameter($Switch) | out-null
                }
                $PowershellThread.RunspacePool = $RunspacePool
                $Handle = $PowershellThread.BeginInvoke()
                $Job =  [pscustomobject][ordered]@{
                    Handle = $Handle
                    Thread = $PowershellThread
                    object = $ItemObj.ToString()
                    Started = Get-Date
                }
                $Jobs += $Job
            }
        #}
    }
    End{
        $GlobalStartTime = Get-Date
        $continue = $true
        While (@($Jobs | Where-Object {$_.Handle -ne $Null}).count -gt 0 -and $continue)  {
            ForEach ($Job in $($Jobs | Where-Object {$_.Handle.IsCompleted -eq $True})){
                $out = $Job.Thread.EndInvoke($Job.Handle)
                $out # return vers la sortie srandard
                #Write-Host $out -ForegroundColor green
                $Job.Thread.Dispose() | Out-Null
                $Job.Thread = $Null
                $Job.Handle = $Null
            }
            foreach ($InProgress in $($Jobs | Where-Object {$_.Handle})) {
                if ($TimeOutGlobal -and (($(Get-Date) - $GlobalStartTime).totalseconds -gt $TimeOutGlobal)){
                    $Continue = $false
                    #Write-Host $InProgress -ForegroundColor magenta
                }
                if (!$Continue -or ($TimeOutThread -and (($(Get-Date) - $InProgress.Started).totalseconds -gt $TimeOutThread))) {
                    $InProgress.thread.Stop() | Out-Null
                    $InProgress.thread.Dispose() | Out-Null
                    $InProgress.Thread = $Null
                    $InProgress.Handle = $Null
                    #Write-Host $InProgress -ForegroundColor red
                }
            }
            Start-Sleep -Milliseconds $SleepTimer_ms
        }
        $RunspacePool.Close() | Out-Null
        $RunspacePool.Dispose() | Out-Null
    }
}

오래된 스레드이지만, 이에 대한 저의 기여는 실행 중인 작업을 세는 부분입니다.위의 답변 중 일부는 0개 또는 1개의 실행 중인 작업에 대해 작동하지 않습니다.제가 사용하는 작은 속임수는 결과를 강제 배열로 던지고 세는 것입니다.

[array]$JobCount = Get-Job-State Running

$JobCount.Count

2023년 답변은 다음과 같습니다.

$list = invoke-sqlcmd 'exec getOneMillionRows' -Server...
$list | % -Parallel -ThrottleLimit 8 {
    GetData $_ > $_.txt
    ZipTheFile $_.txt $_.txt.zip
    ...
}

ForEach-Object cmdlet은 Powershell 7.0에서 여러 프로세스를 병렬로 시작할 수 있는 기능을 얻었습니다.https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/foreach-object?view=powershell-7.3 을 참조하십시오.

언급URL : https://stackoverflow.com/questions/8781666/run-n-parallel-jobs-in-powershell