The router is 192.168.68.74 and the Asus is 192.168.68.64. The Subnet on the router is 255.255.252.0 and the Gateway is 192.168.68.1. I believe the Subnet on the Asus is 255.255.252.1 also because I seem to remember that they had to be the same and early on one was 252 and the other 255.
The router and the gateway should be the same
To elaborate that: A subnet mask of 255.255.252.1 is invalid so perhaps you have mis-remembered it (a lot of kit would not have allowed you to enter such a subnet mask, but some will.) In an IP network (or "subnet" as we often refer to them) every device should be using the same subnet mask, so change the one on your ASUS to match everything else which prime-facie looks like it is 255.255.252.0
Subnet Mask 255.255.252.0 is unusual for small SOHO networks, (255.255.255.0 is more common) but it should be fine as long as everything is using the same one.
IP addresses are structured, a bit like real world postal addresses, and the subnet mask tells stations how to interpret that structure. Using the postal address analogy again, IP addresses consists of a "street-name"+"house-number" and the subnet mask tells hosts how much of the address is "street-name" and how much is "house-number" using something called a binary bit mask.
A subnet mask of 255.255.252.0 will avail an address space for 1022 host stations which is way more than most SOHO users need, but if it's working I'd leave it alone. (255.255.252.0 means 192.168.68.0 is the "subnet address" ("street-name") and 192.168.68.1 thru 192.168.71.254 are the station addresses ("house-numbers") with 192.168.71.255 being reserved for a special "broadcast address" ie "all stations.")
A subnet mask of 255.255.252.1 is completely invalid and I would be surprised if it works at all.
In one "knows what one is doing" we can use subnet masks to achieve certain effects, but a simple SOHO network rarely needs such features, so we just need to use it to instruct all stations to interpret the IP address structure in the same way, ie give them all the same subnet mask and default gateway (of which more below.)
When an IP host has an IP data packet to send, it uses it's IP address and subnet mask to determine whether the "destination" host is on "the same" subnet as itself (same "street-name") or a "different" subnet (different "street-name"). If the "the same" then the sending host will using some magic that send the traffic directly across the local network. If "different" the sending host will send the traffic to it's configured "default gateway." The "default gateway" device on receiving any traffic will determine where to send the traffic next to move it towards it's destination. This is what IP Routers "do" - make routing decisions about where to send any IP traffic they receive based on the traffic's destination IP address and a table of "routes" each router builds internally.
Thusly, in the SOHO use case, the "default gateway" for all hosts should be the IP address of your Internet connected router as that's the only "route" outside the local network that's of any concern to you (ie "the rest of the world.")
If your ISP Router has an IP address of 192.168.68.74 and any of your hosts have a default gateway of 192.168.68.1 then this is incorrect. Either your ISP router's LAN IP address is wrong (ie it should be 192.168.68.1) or your default gateway settings are incorrect (ie it should be 192.168.68.74.) You should verify what your ISP connected router's "LAN" IP address is and that should be the "default gateway" for all other stations.
When looking at router configs, be aware that routers (and only routers) may also have something called a "default route." This is a different "thing" to the "default gateway" despite the names being similar. Don't mess with the default route in your ISP router or you may break the whole network's Internet access (it will probably have self configured it in any case and may no let you change it.) Your ASUS should not need a default route, but it won't hurt if it has one. Since we don't want the ASUS to route any traffic (irrespective of whether it's in AP mode or router mode) it doesn't matter what it's default route is. But the ASUS LAN IP address and subnet mask do matter and need to be consistent with the rest of your subnet as discussed above.
Also, in AP Mode your ASUS may need to be told a "default gateway" but just set that the same as the rest of your network (ie 192.168.68.74 or 192.168.68.1 when you have determined which of those is the correct IP address you ISP router.) In "router mode" your ASUS (probably) won't have a "default gateway" as such things are meaningless to routers (routers have "default routes" instead.)
Whatever "mode" you set the ASUS to, you must ensure it's built in DHCP Server is "off." AP mode probably turns it off for you automatically. In Router mode, you may need to do it yourself.
I cannot think why the media server functionality should differ between the ASUS AP Mode and Router mode, but it depends how ASUS wrote their software.
However, fix the IP addressing first, them we can investigate what's going on with SMB and file serving.
As described, it's sounds more complicated than it is, especially to someone new to all this, but once you've got the IP addressing sorted, you may find it all "magically" just works.