How many resources can I request?
How many resources can I request?
The maximum available CPU, GPU, memory, and disk depend on your deployment’s compute pools. To see the pools and the resources they provide, select Compute in the left-hand navigation and click Pools.If you request more resources with To resolve the error, lower the resources requested in
@resources than any pool can provide, the task fails with an error:@resources, or contact your administrator to add more compute capacity.What is the maximum number of items that can be processed with foreach?
What is the maximum number of items that can be processed with foreach?
A
foreach can iterate over any Python list, potentially containing hundreds of thousands of items.To guard against launching an excessive number of tasks by accident, Metaflow limits the number of splits with the --max-num-splits flag. To run a wider foreach, increase the value. For example, --max-num-splits=10000.Metaflow also limits how many tasks run concurrently with the --max-workers flag. Increasing --max-workers speeds up processing through more parallelism, at the cost of additional load on the cluster. To watch the load, select Compute in the left-hand navigation and click Pools.