I gave LB 4G ram, however it took more than the jvm argument(6.8G).
/usr/lib64/openjdk-8/jre/bin/java -Xincgc -XX:-UseAdaptiveSizePolicy -XX:-OmitStackTraceInFastThrow -Xmn128m -Xmx4096m ...
2ab8a685-585a-43a6-be60-6764eb6e41cb-image.png
84e7da06-6a44-43d6-9375-dcf68eca5e6f-image.png (Video Settings-> Other Settings -> Lagometer(optifine))
Idk what to blame, either it is my self compiled jvm has sth wrong or LB has a memory leak or a problematic gc. Or both lol.
Edit: with no other option/tuning -XX:+UseG1GC performed worse than -Xincgc, mem usage quickly got to 5G and the FPS lower than -Xingcgc(tho the gc frequency was significantly lower than -Xingcgc).