Kartik Bali

PhD Candidate · Helmholtz-Zentrum Hereon & TU Hamburg

I research how vision-language models understand and act in the 3D world: deciding where to look, grounding language in geometry, and turning what they see into executable programs (CAD, BIM). In short, I teach AI to look at 3D objects and understand them the way an engineer would. Along the way I work on RL post-training with execution rewards and on probing what these models actually rely on. I collaborate closely with DFKI.

Before my PhD, I did my M.Sc. in Computational Science & Engineering at the Technical University of Munich, with a master thesis at BMW on physics-guided generative models, supervised by Prof. Nils Thuerey. I hold a B.E. in Mechanical Engineering from BITS Pilani, Hyderabad. Outside research, I play lead guitar and sing.

I am looking for research internships in 2027. Feel free to reach out!

Email CV Google Scholar GitHub LinkedIn
Kartik Bali

News

Research

First-author papers

GReFEM pipeline

GReFEM: Multimodal LLMs as Zero-Shot Semantic Assistants for Physics-Guided 3D Mesh Refinement

Kartik Bali, Mahish K. Guru, Christian J. Cyron, Roland Aydin

NeurIPS 2026 ACCEPTED

Can off-the-shelf multimodal LLMs serve as zero-shot geometric proxies for physics? We ask MLLMs to localize stress-critical regions from physics-guided prompts, bridging 2D pre-training and 3D geometry with orthoViews, a trained view-selection module that maximizes feature observability. Without any FEM solve or simulation training data, GReFEM closes 58% of the gap between the unrefined mesh and a reference-solve oracle at a matched element budget, and we map where MLLMs succeed and where they still fall short.

Paper Code
MV-GEL overview

MV-GEL: Language-Driven Multi-View Geometric Entity Localization on Meshes

Kartik Bali, Roland Aydin

ECCV 2026 ACCEPTED

Grounding fine-grained geometric entities (edges, planar regions, curved surfaces) on 3D meshes from natural language. Prompt-conditioned view selection over a VLM reasoning-segmentation backbone, lifted to the mesh with geometry-aware ray casting. Up to 1.7× face-level IoU and 4.5× edge-level F1 over CLIP-based and random-view baselines.

Paper Code
SightCAD joint training SightCAD reconstructions: ground truth (top) vs ours (bottom)

Look Where You Can: Active View Selection for CAD Reconstruction under Occlusion

Kartik Bali, Mahish Guru, Yiderigun Borjigin, Alexandra Starostina, Christian J. Cyron, Roland Aydin

Preprint, 2026 UNDER REVIEW

Real objects are scene-embedded: walls, floors, and corners make much of the view sphere unreachable. A learned view selector picks feasible views for a VLM that writes executable CadQuery code, and both are trained jointly with RL against execution reward. Outperforms strong baselines by +21 / +17 effective-mIoU on occluded DeepCAD / Fusion360, transfers to real T-LESS and MP6D images, with ≤1.5% invalid code. Layer-wise ablations of each view's visual tokens give a mechanistic account of which views make which program parameters legible to the VLM. (Hover: ground truth vs. our reconstructions on DeepCAD, Fusion360 and T-LESS.)

Paper

Ongoing

Point cloud to BIM program

Point2BIM: From Real Laser Scans to Executable Building Programs

Kartik Bali, in collaboration with DFKI

ONGOING

Turning real terrestrial laser scans of buildings into executable BIM programs (exported to IFC4) with vision-language models. Training with SFT followed by GRPO on execution-based geometric rewards, and targeting generalization to real scans rather than synthetic scenes.

Co-authored

  1. J. Minder, V. Moskvoretskii, R. Singhal, D. Jiao, A. Arditi, S. Cui, Y. Borjigin, K. Bali, S. Krsteski, H. Raj, H. Nguyen, J. Brinkmann, A. Anderson, R. Aydin, R. West. Synthetic Persona Pretraining: Alignment from Token Zero. Preprint, 2026. [arXiv]
  2. M. K. Guru, V. Gupta, K. Bali, et al. From Finite Element Analysis to Self-Optimization: FRAME (Finite element Reasoning and AI-Agent Model Engine) and the Era of Agentic AI in Engineering. Under review at npj Computational Materials, 2026.
  3. M. Tacke, …, K. Bali, et al. Constitutive Scientific Generative Agent (CSGA): Leveraging Large Language Models for Automated Constitutive Model Discovery. Machine Learning for Computational Science and Engineering, 2025.
  4. B. List, L.-W. Chen, K. Bali, N. Thuerey. Differentiability in Unrolled Training of Neural Physics Simulators on Transient Dynamics. Computer Methods in Applied Mechanics and Engineering (CMAME), 2025. [arXiv]
  5. K. Bali, K. Supradeepan. Numerical Analysis of Flow Past an Oscillating Circular Cylinder using Arbitrary Lagrangian Eulerian Consistent Flux Reconstruction Scheme. 2019. [arXiv]

Experience

Earlier Projects

Generative Geometry Design for CFD

Master thesis (TUM × BMW): physics-guided generative models synthesize low-pressure-drop flow manifolds, with fluid topology optimization in PhiFlow.

Slides
Dense Point Clouds from a Single RGB Image

Two-stage point set generation with an encoder-predictor and folding decoder (ML for 3D Geometry, Prof. Angela Dai).

Report
3D Shape Completion with DeepSDF & 3D-EPN

Continuous SDF shape representations on ShapeNet: completion and latent interpolation.

Code
Object Detection with LiDAR Fusion

YOLOv3 and SSD on KITTI; L2-norm pruning for faster SSD inference and early LiDAR fusion.

Code
GNNs for the Kuramoto–Sivashinsky Equation

Message-passing GNNs and CNNs with temporal unrolling for chaotic PDEs; led to the CMAME 2025 paper.

Paper
Topology Optimization with RL in JAX-FEM

Double-Q agent with JAX-FEM as the environment (TUM Data Innovation Lab).

Report
Sphere Packing via Differentiable Physics

Gradient-based minimization of overlap energy to find optimal 2D packings in PhiFlow.

Code
Pedestrian Distributions with VAEs

Learning crowd distributions in the TUM MI building for evacuation scenarios.

Code
Interactive 3D Ray Tracer

First-person ray-tracing app in C++ with SDL2.

Code