aan asked me some great questions in a comment recently and I'm posting my response as a full blog post to be able to elaborate a bit more.

Preparation

Coastlines are one of the most defining features of cartography, and in many ways even more so when you work in 3D, so they need a lot of steps and input to look good in the end. In most cases there are already some form of map or description of the area that the final map need to adhere to, meaning follow more or less strict.  How strict depends on how the existing map or text portray it compared to the result needed. If the goal is to create a realistic model of the terrain and what we have are a vague text or an illustration that looks cool but very unrealistic, a lot of changes needs to be made. In other cases the problem is the opposite, you might get a picture of a terrain for a commission and the goal is to use that as a base for something that should be part of an illustrative fantasy looking parchment map. In my work the former is the most common case, but both ends or cases in between occur all the time.

When I have decided the goal and gathered the sources, the next step is to prepare a useful guide image from the material.  This usually means using Photoshop to get rid of things like text symbols and things that are clutter and in the way. The previous map might be done in some artistic perspective so stretching and warping it try try and create a top down view are needed. This process can range from very easy to very complex depending on the quality and size of the source.

If all I have are text I read it all and take geographical notes, and then sketch out a simple terrain and then read the text again to make sure that all the geographical mentions in the text matches my sketch. Then I use imagination to try and sketch out the rest of the terrain to fill out the gaps keeping in line with the theme and spirit of the story. The overall environment of the story is key for this, is the story set in a jungle, mountains, what is the climate, is it a demonic waste land, or a idyllic farmland and so on.

Using the sketch I can then use that as an overlay when working in World Machine or Gaea to create the terrain. I'm only using it for visual guidance, tweaking the fractals to generally follow the guide image coastlines. It is possible to create a mask (a black and white or greyscale image) in Photoshop and use that directly in terrain creation. This have a strong tendency to limit the fractals creating harsh unnatural, sterile coastlines, but in some cases especially with large scale maps it can work well. Vectors are possible to import and use in World Machine but they are tricky to get right and require even more work to get detailed.

Create

Yes, I use World Machine or Gaea to try and sculpt terrain to match the guide image. It sounds simple but can be a true pain, fractals live their own life and you need to terrain features like hills, rivers and erosion, terracing an more to also work together to create a good looking terrain that meats the specifications. The coastlines are only one of the parameters to get right, import yes, but only one of many things to make a good looking terrain.

The key is to try and understand how the different aspects of the process like the fractal generating the elevation, erosion, filters like Terracing, Glaciation, Displacement and Flow Restructuring, each affects the result so you can tweak the the right way. Then you have the different forms of Erosion that are often needed to make the terrain look realistic, like old weathered rough low hills for that haunted moor, or wind swept rocks for a sandy desert. Costal Erosion is especially critical for coastlines, forming long beaches of flattened terrain and shallow water aplenty, but that needs to be applied sparingly, far from all coastlines are pristine beaches.

Islands and, bays and inlets are features that I try to program into the cocktail of fractal generators, filters and erosion used to create the terrain.  You use shapes in World Machine to apply the various effects and there is a setting that lets you apply a "warping factor" as well as a fall off helping you to create the endless varying of the same theme across large areas that we see all over the natural world. The key is to blend a lot of different factors together and let probability and randomness work out the details for you. It is a trial and error process, but over time you learn and you can re use combinations that worked well in the past.

The key is to study the real work, imagine what a fantasy world could look like and then use a systematic trial and error to try and achieve your vision.

Other ways of creating coastlines

The way I described it above is the way I go about creating coastlines for 3D terrain. That might not be what you need or want to create, which means you can use different approach using Photoshop, texture generators, or even pancakes!

What you need are some fractal large scale random pattern, that can come from almost anywhere, real world data, patterns in nature, random scribbles, or the burn marks on pancakes. Make pancakes in a frying pan, use your smart phone to take a top down image of each pancake. Use filters in Photoshop (or GIMP) to reduce the image down to pure black and white and you have a set of coastlines, islands or lakes as needed. Combine, resize and rotate and you can create any type of realistic set of coast lines for any type of project. Randomness, creativity and base knowledge of a image editor software is what is needed along with a vision of what the goal is for what you need to accomplish.

Lots of things besides pancakes can be used for this, clouds, splash patterns, a rock wall, bark on a tree etc. Let nature inspire you!