There can be two types of duplication of rows in a table
1. Entire row getting duplicated because there is no primary key or unique key.
2. Only primary key or unique key value is different, but remaining all values are same.
Scenario 1: Delete duplicate rows without primary key or unique key.
Let us create the following example.
create table customers1 (CustId Int, CustName Varchar(20), CustCity Varchar(20), Passport_Number Varchar(20)) go
Insert into customers1 Values(1, 'John', 'Paris', 'P123X78')
Insert into customers1 Values(2, 'Martin', 'London', 'L873X92')
Insert into customers1 Values(3, 'Smith', 'New York', 'N293Y99')
Insert into customers1 Values(1, 'John', 'Paris', 'P123X78') go
select * from customers1 go
We want remove one of the duplicate records of John.
By issuing the following summary query, we can see which see which records are duplicate.
select * from customers1 Group by Custid,CustName, CustCity, Passport_Number Having count(*) > 1
Now we will add this row to a local temporary table.
Select * into #Temp_customers1 from customers1 where 1 = 2 Insert into #Temp_customers1 select * from customers1 Group by Custid,CustName, CustCity, Passport_Number Having count(*) > 1
Now the situation is that the duplicate row is in the local temporary table. All we need to now is to delete records from main table customers1 as per matching custid of the local temporary table.
Delete from customers1 where custid in (select Custid from #Temp_customers1)
Will the above query work? Not entirely, as by using the above query, we lost all the duplicate records!! Let us see the table again.
select * from customers1 go
Now to keep one record of John, we will take help of the local temporary table again. Let us add the same record from temporary table into customers1 table.
Insert into Customers1 select * from #Temp_customers1 go
Finally we got a single record of John at the end. Let us confirm by seeing the Customers1 table.
select * from customers1 go
Once done, we can drop the local temporary table.
Scenario 2: Delete duplicate rows where primary key or unique key value is different but remaining values are same.
Let us create the following example.
create table customers2 (CustId Int Primary Key, CustName Varchar(20), CustCity Varchar(20), Passport_Number Varchar(20)) go
Insert into customers2 Values(1, 'John', 'Paris', 'P123X78')
Insert into customers2 Values(2, 'Martin', 'London', 'L873X92')
Insert into customers2 Values(3, 'Smith', 'New York', 'N293Y99')
Insert into customers2 Values(4, 'John', 'Paris', 'P123X78')
Insert into customers2 Values(5, 'John', 'Paris', 'P123X78')
select * from customers2 go
Here is the same customer’s record, but this time John’s record has been added thrice with different customer ids but same Passport number!
Scenario 2.a: Delete Duplicate rows but keep one using CTE
We need to use the technique of Self Join initially to check for duplicate records containing different custid but same passport number.
select distinct a.* from customers2 a join customers2 b on a.custid <> b.custid and a.CustName = b.CustName and a.CustCity = b.CustCity and a.Passport_Number = b.Passport_Number
Now we have realized that custid 1, 4 & 5 are duplicate. The self-join statement accompanied by delete statement will give us the desired output of keeping the last duplicate record by eliminating all the previous duplicate records. We will use the Common Table Expression (CTE) and put the Self Join query in it.
With Duplicates as (select distinct a.custid as Customer_ID from customers2 a join customers2 b on a.custid <> b.custid and a.CustName = b.CustName and a.CustCity = b.CustCity and a.Passport_Number = b.Passport_Number ) Delete from Customers2 where custid in (select Customer_ID from Duplicates) and custid <> (select max(Customer_ID) from Duplicates)
Let’s check which rows got deleted.
select * from customers2 go
Scenario 2.b: Delete all duplicate records but keep the first original one
Let’s first truncate the customers2 table and add the same rows again.
Truncate Table customers2 go
Insert into customers2 Values(1, 'John', 'Paris', 'P123X78')
Insert into customers2 Values(2, 'Martin', 'London', 'L873X92')
Insert into customers2 Values(3, 'Smith', 'New York', 'N293Y99')
Insert into customers2 Values(4, 'John', 'Paris', 'P123X78')
Insert into customers2 Values(5, 'John', 'Paris', 'P123X78') go
The only change in the sub query will be that we need to use min(CustomerID) instead of max(CustomerID).
So the query will be as follows.
With Duplicates as (select distinct a.custid as Customer_ID from customers2 a join customers2 b on a.custid <> b.custid and a.CustName = b.CustName and a.CustCity = b.CustCity and a.Passport_Number = b.Passport_Number ) Delete from Customers2 where custid in (select Customer_ID from Duplicates) and custid <> (select min(Customer_ID) from Duplicates)
Let us confirm this in the customers2 table.
select * from customers2 go
And that’s how we can delete duplicate records in SQL Server with tables without primary key, containing primary key and by keeping one original row.