Figure above shows boost converter with 5V input and 50V output.
I chose voltage input as 3.7V LiPo battery and 5V booster(MP2636 https://www.dfrobot.com/product-1613.html).

Whenever I connect 5V booster to boost converter above, It shuts down.
I have no idea whether shutting down is caused by protection circuit on battery or MP2636.
So I decided to limit input current of boost converter as 2A.

Can I get some help about this problem?

 

Limit converter out current using circuit below.
R1new(Ohm) = 660 / Iout_limit(mA)

 

I have no idea whether shutting down is caused by protection circuit on battery or MP2636.

You can monitor the 5V so see if that's the problem rather than guessing.

But it's poor design, expensive, and gives poor efficiency to boost the 3.7V battery twice.
Design the boost circuit to directly increase the battery voltage to 50V.
The lm2577 can operate down to 3.5V input which should use at least 80% of the battery capacity before stopping ( for best life, you generally don't want to discharge a li-ion battery much below that voltage anyway).