There are many potential knowledge yielding systems, each of which begin with certain premises, and at least some of those premises are untestable. For instance, the scientific method itself cannot be verified with the scientific method.
Why we have confidence in science
We tend to have high confidence in science, generally, because:
Scientific principles are frequently tested
Ideally, the information (data and models) are tested and re-tested in numerous orthogonal ways, usually in a manner designed to minimize bias. Principles which are newer or controversial tend to be tested the most.
Many of our scientific theories are integrated into other technologies and hence are implicitly tested repeatedly by many independent observers.
For instance, the scanning tunneling microscope operates on (and implicitly tests) quantum theory (specifically, the exponential wave decay of electrons through a barrier), atomic theory (when the atomic properties of atoms, like atomic size and packing, are measured), and electric current. Nothing compels the instrumentation to conform to previously discovered scientific models except the underlying reality.
Scientific theories are subject to being falsified
If you cannot conceive of a result which would falsify a scientific theory, then we do not call it science but philosophy or something else.1 A good scientific experiment involves genuine risk that a theory may be disconfirmed depending on the results.
Scientific principles may be independently verified
Obviously, some experiments are easier for the average person to perform than others, but a motivated person can, in theory, test the validity of any scientific theory or finding they wish.2
From religious systems
Information Theory Substantiation
The father of information theory viewed information as “the resolution of uncertainty”, and he
A great discussion on the principle of falsifiability is found here. And even scientific theories that are used to explain events retrospectively (like the Theory of Common Descent), are daily subject to falsifiability on thousands of fronts. A great example of this can be found in the “potential falsification” sections of the 29+ evidences for macroevolution site. ↩
We can, in principle, test any scientific theory we desire. For example, if we take for granted the data on the distribution of muons in the atmosphere and muon decay rates, then we can construct a cloud chamber for $100 and verify the theory of relativity. ↩