Check out all the absolute nightmare cave systems we have on earth right now, not just places full of lava, but things like: caves full of odourless toxic gas, or simply with low or no oxygen; caves full of water where simply touching the walls causes the water to silt up so badly you can't see your own hands and lose your sense of direction; massive crevasses that take hours to scale down that require the climber to bend over backwards to keep their footing, and are dangerous with modern climbing equipment; crawl spaces so narrow you can't turn around and have to push your kit ahead of you or drag it behind you, crawl spaces several 1000m from the nearest entrance, where getting stuck won't just kill you but trap everyone in front and behind you in the cave; caves where even the slightest increase in moisture or temperature turns the ceiling and walls to sponge, and risks collapsing the whole structure; invisible lakes of crystal clear water cold enough to cramp you up and drown you in seconds; dead ends; narrow ledges that can only take the weight of one person at a time without crumbling.
Caves are lethal without magic and monsters, combining the above with the kind of underground horrors of dungeon crawling and you can make things truly challenging.
To avoid making things just a skill check or two focus on the tools required to navigate a problem not just the roll required. How could the party scale the ravine, or descend a sinkhole, or squeeze through a crawl space? Make failures increase tension rather than stall it; a failed perception check to find hand holds causes a character to fall and now they're hanging from the rope pulling it taught, the rope is now fraying as its rubbing on the wall, the party only have so much time before it snaps... what do they do?
These ticking clocks keep the tension, and require solutions not just dice rolls, how do they get their friend back to the wall? how do they prevent the rope from breaking? the dice rolls are only there to figure out if their plan works if there's a chance it doesn't.
And for some of these challenges they will solve them without a roll, how do they get their heavy kit across a fragile ledge, a bag of holding. How do they climb to the top of a tall shaft, a rope of climbing, how does the Wizard scale the ravine, a spider climb or a fly spell.
In some of these cases you can add a complication if the pacing isn't working. Everyone having darkvision is great for solving the lack of light, but there's a monster in here that's a real threat thanks to the terrain but it's afraid of bright light but concentrating on light spells means the casters can't concentrate on fly or spider climb.
Hope this all helps.