One way might be to:
build an intermediate table that uses the pivot data to show the filtered data and then you make the chart on that table. The values not selected should result in N/A so not to plot them on the chart.
Basically your number of series remains the same. Thus the width will not change on a slicer selection.