I settled with something really simple called hemispherical lighting, which basically means to have separate up and down (sky and ground) colors and then blend them depending on the normal of the surface. My first idea was to use cubemaps to do something similar, but since the cubemap needs to be very blurred, using hemispherical lighting gave pretty much the same result and is a lot faster.
Next up was the direct lighting from the sun and this was quite simple. I could just add some tweak to the existing shaders and make it work. Basically sunlight is just like normal light but without any position or attenuation. This means every pixel is lit from the same direction and strength is independent of distance to the source (as it does not have a position.