split a set of tables up into equalish size groups

problem: transfer a set a tables from one server to another, 4 tables in parallel, using a pre-created stored proc, they should all complete around the same time

solution: within a dataflow create a source which fetches your list of tables and each tables size, include a row number field order on size in descending order, put this in a cte.

Select from the cte and create a new column as follows 1+(((r%4)+(CEILING(r/4.0))) % 4) called group_id, this ensure that each group is of a similar size, I cant explain why it works, it made sense as I was writing it, it just works.

Use a conditional split to send each group to an ole db command operator each of which executes your import procedure and passes the relevant parameters

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: